Yep, you can just drift it out. But I'm puzzled why it's stuck... I have a '47 rigid and a '51 Anstey and neither were stuck like that...
Don't hit it too hard. The brake lug is pegged and brazed on (only), and it would/could bend. Extraction would be better, using something like a track rod end splitter.
(AOMCC Exec Committee & Selly Oak branch rep)
'47VB, '51KH, '58FH, '05 BMW oilhead boxer.