Swift arrays are not necessarily contiguous blocks of memory (although they may be in some cases so some code may work in some cases but not in others). Some operations may trigger them to be converted to a contiguous block of memory and I believe that is what can particularly cause issues when there is access from multiple threads.
[Update - This post is more popular than I expected (thanks Reddit). I've added some additional notes at the bottom. Also the previous post about optimisation is probably more interesting and valuable.]
Even the code as described here may not work (and I haven't tested at all) when there are multiple reads or writes that access the same elements of the array. All the code I used was either reading from a constant array (safe from multiple threads/queues) or writing into non-overlapping ranges with no reads occurring (only seems safe with the technique described here).