Thousands had marched to the border at the start of a six-week protest, dubbed the Great March of Return.

The Israeli military said soldiers had opened fire after rioting.

UN Security Council members meeting in New York have called for an investigation into the violence.

Palestinians have pitched five camps near the border for the protest. They are demanding that refugees be allowed to return to homes that are now in Israel.

Palestinian President Mahmoud Abbas urged the UN Security Council to help provide "protection for our Palestinian people".

"I... place full responsibility on the Israeli authorities for the loss of the martyrs who were killed today," he said.

The Israel Defense Forces (IDF) said there were about 17,000 Palestinians in five locations near the border fence. It said it had "enforced a closed military zone" in the area surrounding Gaza.

Although most protesters stayed in the encampments, some groups of youths ignored organisers' calls to stay away from the fence and headed closer to Israeli positions.

The IDF said troops were "firing towards the main instigators" to break up rioting that included petrol bombs and stones being thrown at the fence.

A spokesman said all those who were killed had been trying to breach or damage the border fence, the Jerusalem Post reports.

Israel said it had targeted sites of the Hamas militant group.

Israel deployed tanks and snipers. Witnesses said a drone had been used in at least one location to drop tear gas.