There is a setting you can enable that prevents applicants from being able to apply to closed jobs. In your Settings menu, go to Company Preferences. The preference you are looking for is Disable Closed Job View. Make sure this is turned on to prevent applicants from applying to closed jobs.

Another possible cause is to make sure the job has been removed from the web. Navigate to the Job profile, specifically the Web Description Tab. Make sure the button in the top right of this tab reads "Publish to Web". If it reads "Remove from Web", click this button to remove it.