The Brother from another Planet
(1984)

Joe Morton is an escaped alien slave who crashes on Ellis Island, and settles in Harlem. 

Despite being mute and only having 3 toes, he finds a place to live, employment and love. 

Two "men in black" bounty hunters (John Sayles and David Strathairn) track him down but he is saved by other escaped alien slaves.  
 

Thanks Don P.!