The total beat the 260 set by Sri Lanka against Kenya in 2007.
All-rounder Glenn Maxwell hit 145 not out from just 65 balls.
Sri Lanka - who had won the toss and elected to bowl first - could only manage 178-9 in reply as Australia won the first game of the two-match series by 85 runs.
Maxwell, who opened the batting, hit nine sixes and 14 fours in his innings - the second-highest in international T20 cricket.
Australia team-mate Aaron Finch holds the individual record score of 156, which came against England in 2013.
However, 27-year-old Maxwell was only playing as an opener after Finch broke a finger during the recent one-day series against Sri Lanka.
Australia's T20 record comes just eight days after England set a new record score in one day international cricket, making 444 against Pakistan - which also beat a record previously held by Sri Lanka.