Walking out during Yizkor

Over the festival of Shavuot, I found the Yizkor service a very moving occasion. It wasn't only the memories of my father, sister and grandfather, but also the memorials for Jewish martyrs, soldiers and even Chief Rabbis. I found myself thinking of my father's father, who escaped with his family from Eastern Europe before the Nazis invaded, and of my father's cousins, who stayed behind and perished in the Shoah.

With this in mind, I have never quite understood why so many walk out during this part of the service. I will never forget when we lived in Israel back in 1983, I walked out during Yizkor and was shocked to see the Rabbi outside too, chatting and laughing with other congregants. I found it all rather unrabbinic. The service was in progress, and here was the Rabbi, joking with friends outside.

I know that the tradition of walking out before Yizkor is partly to do with superstition (despite the fact that Judaism frowns upon it), but I think it probably has a lot more to do with the fact that it's a good opportunity to catch up with friends without feeling guilty about skipping the service. Isn't it about time our Rabbis spoke out against this nonsense? Why should the sole responsibility for remembering martyrs, soldiers, rabbis and even lost friends only fall on the people who have lost their relatives? Isn't there an obligation for all of us to remember? 'kol yisrael arevim zeh lazeh.'