The planet, more than four times the mass of Earth, is one of three that the team detected around a red dwarf star called Wolf 1061.
"It is a particularly exciting find because all three planets are of low enough mass to be potentially rocky and have a solid surface, and the middle planet, Wolf 1061c, sits within the 'Goldilocks' zone where it might be possible for liquid water - and maybe even life - to exist," said lead study author Duncan Wright, from University of New South Wales in Australia.
"It is fascinating to look out at the vastness of space and think a star so very close to us - a near neighbour - could host a habitable planet," Wright said.
"While a few other planets have been found that orbit stars closer to us than Wolf 1061, those planets are not considered to be remotely habitable," Wright said.
The three newly detected planets orbit the small, relatively cool and stable star about every 5, 18 and 67 days. Their masses are at least 1.4, 4.3 and 5.2 times that of Earth, respectively.
The larger outer planet falls just outside the outer boundary of the habitable zone and is also likely to be rocky, while the smaller inner planet is too close to the star to be habitable.
The team made the discovery using observations of Wolf 1061 collected by the High Accuracy Radial Velocity Planet Searcher (HARPS) spectrograph on the European Southern Observatory's 3.6 metre telescope in Chile.
"Our team has developed a new technique that improves the analysis of the data from this precise, purpose-built, planet-hunting instrument, and we have studied more than a decade's worth of observations of Wolf 1061," said Chris Tinney, head of the Exoplanetary Science at UNSW.
"These three planets right next door to us join the small but growing ranks of potentially habitable rocky worlds orbiting nearby stars cooler than our Sun," said Tinney.
Small rocky planets like our own are now known to be abundant in our galaxy, and multi-planet systems also appear to be common.
However, most of the rocky exoplanets discovered so far are hundreds or thousands of light years away. An exception is Gliese 667Cc which lies 22 light years from Earth.
It orbits a red dwarf star every 28 days and is at least 4.5 times as massive as Earth.
"The close proximity of the planets around Wolf 1061 means there is a good chance these planets may pass across the face of the star. If they do, then it may be possible to study the atmospheres of these planets in future to see whether they would be conducive to life," said Rob Wittenmyer of UNSW.
The study will be published in The Astrophysical Journal Letters.