Rio blamed the move on problems of power supply terms.
Cahya Mata Sarawak (CMS), which owns 40% of the Sarawak Aluminium Company (Salco) smelter development in Sarawak state, announced the termination to the Malaysian stock exchange.
Aluminium has been a thorn in the side of Rio since its 2007 $38.1bn purchase of Alcan and it wrote down the division by $8.9bn last month.
Rio Tinto will close its Lynemouth smelter in UK this month, costing 300 jobs, and is believed to be considering the future of its Tasmanian smelter where 600 people work.
Rio and Cahya Mata "agreed that they would cease to pursue plans to jointly develop an aluminium smelter at Samalaju in Sarawak but remain open to other future possible collaborations".
The smelter was expected to have an initial production capacity of 550kt/y of aluminium, with the potential to expand to 1.5Mt/y.