Yep, you can just drift it out. But I'm puzzled why it's stuck... I have a '47 rigid and a '51 Anstey and neither were stuck like that... Don't hit it too hard. The brake lug is pegged and brazed on (only), and it would/could bend. Extraction would be better, using something like a track rod end spl...