Simitis' warning came on the heels of a fatal accident Monday at the site. To date, four workers have died during the construction of the village, and some workers and government officials worry that construction delays are causing companies to take safety shortcuts.
Simitis' warning about workplace safety came after Olympic Village officials said they would take legal action against construction companies that do not follow regulations.
There are hundreds of workers at the Olympic Village site, one of the few projects that is ahead of schedule. About 1,500 workers called for a strike Monday after a coworker was killed when he fell off a crane.
A government official said 85 percent of the workers building the village are unskilled and said their employers are to blame for not providing adequate training and safety measures.
GSSE, Greece's largest labor union is supporting the strike despite the fact that many of the workers are from foreign countries.
"GSEE denounces this unacceptable situation that costs human lives, especially at a project that concerns the Olympic Games," the union said in a statement.