Why do we have linkers for different architectures? Service of the linker is to resolve addresses. So how it is related to the instructions of target architecture?
Different architectures have different address formats in their instructions which the linker has to know in order to manipulate them.
Relative addressing may lead to different instructions depending on the size of the relative address.
More complicated schemes also exist, for example for ARM.
Usually it is described in a supplement to the specification of the linker format, for example take a look at the documents linked from this Wikipedia article on the ELF format.
There are many, many reasons, and I cannot exhaustively list all of them.
Linkers need to do a whole lot more work when an architecture does not support position-independent code. In those cases, even jumps within functions need to be resolved.
Linkers need to create architecture-specific headers, such as ELF or PE headers.
Linkers need to include resources, data forks or similar things on platforms that support them
Linkers need to instantiate exported C++ templates
Linkers also need to deal with addresses that cannot be resolved yet. This can include system calls, or dynamically loaded libraries.
When linking a dynamic library, a linker needs to export more than a single entry function. Different architectures uuse different methods to indicate which functions are to be exported
Linkers may need to insert the actual call sequences if they cannot be determined at compile time. E.g. for architectures that support two instruction sets, an "instruction set switch" needs to be inserted whenever the caller and callee differ in instruction set used.
Linker optimizations can be based on architecture-dependent details. E.g. if function calls within a 4KB region are faster, it makes sense to put caller and callee close together.
Inlining across object files can be done, but requires removal of the call setup, callee prolog, callee epilog and return value handling. Those are architecture-specific, so just recognizing them already takes an architecure-specific linker.
To resolve addresses the linker needs at least to know the endian-ness and size of addresses. Some architectures such as x86 real-mode have rather more complex addressing schemes, some embed addresses in the instruction, so the linker may need to know the address or offset field.
Some linkers can be built to understand multiple architectures. For example, I use gnu ld, gdb, binutils, and assemblers for my cross compiler project, http://ellcc.org. I have an assembler specific for each target, but the linker, debugger, and binutils all understand all the processors. The supported processors are pretty varied: ARM, CellSPU, Mips, MSP430, Nios2, PIC16, PowerPC, PowerPC64, Sparc, X86, X86_64.
Some toolchains are built such that a huge portion of optimizations are deferred to link-time, when whole-program information is available. The benefits of inlining, constant propagation, and many other traditional optimizations, are all most applicable when applied to the entire binary, and not just each object.