# Applications of fixed point theorems in the theory of invariant subspaces

## Abstract

We survey several applications of fixed point theorems in the theory of invariant subspaces. The general idea is that a fixed point theorem applied to a suitable map yields the existence of invariant subspaces for an operator on a Banach space.

MSC:47A15, 47H10.

## 1 Introduction

One of the most recalcitrant unsolved problems in operator theory is the invariant subspace problem. The question has an easy formulation. Does every operator on an infinite dimensional, separable complex Hilbert space have a nontrivial invariant subspace?

Despite the simplicity of its statement, this is a very difficult problem and it has generated a very large amount of literature. We refer the reader to the expository paper of Yadav [1] for a detailed account of results related to the invariant subspace problem.

In this survey we discuss some applications of fixed point theorems in the theory of invariant subspaces. The general idea is that a fixed point theorem applied to a suitable map yields the existence of invariant subspaces for an operator on a Banach space.

In Section 2 we consider the striking theorem of Lomonosov [2] about the existence of invariant subspaces for algebras containing compact operators. The proof of this theorem is based on the Schauder fixed point theorem.

In Section 3 we present a recent result of Lomonosov, Radjavi, and Troitsky [3] about the existence of invariant subspaces for localizing algebras. The proof of this result is based on the Ky Fan fixed point theorem for multivalued maps. The idea of using fixed point theorems for multivalued maps in the search for invariant subspaces was first introduced by Androulakis [4].

In Section 4 we consider an extension of Burnside’s theorem to infinite dimensional Banach spaces. This result is originally due to Lomonosov [5]. We present a proof of it in a special case that was obtained independently by Scott Brown [6] and that once again is based on the Schauder fixed point theorem.

In Section 5 we address the existence of invariant subspaces for operators on the Krein space of an indefinite product, and we present a result of Albeverio, Makarov, and Motovilov [7] whose proof uses the Banach fixed point theorem.

The rest of this section contains some notation, a precise statement of the invariant subspace problem, and a few historical remarks.

Let E be an infinite dimensional, complex Banach space, and let $\mathcal{B}\left(E\right)$ denote the algebra of all bounded linear operators on E. A subspace of E is by definition a closed linear manifold in E.

A subspace $M\subseteq E$ is said to be invariant under an operator $T\in \mathcal{B}\left(E\right)$ provided that $TM\subseteq M$, and a subspace $M\subseteq E$ is said to be invariant under a subalgebra $\mathcal{R}\subseteq \mathcal{B}\left(H\right)$ provided that M is invariant under every $R\in \mathcal{R}$. A subalgebra $\mathcal{R}\subseteq \mathcal{B}\left(H\right)$ is said to be transitive provided that the only subspaces invariant under are the trivial ones, $M=\left\{0\right\}$ and $M=E$. This is equivalent to saying that the subspace $\left\{Rx:R\in \mathcal{R}\right\}$ is dense in E for each $x\in E\mathrm{\setminus }\left\{0\right\}$.

The commutant of a set of operators $\mathcal{S}\subseteq \mathcal{B}\left(E\right)$ is the subalgebra ${\mathcal{S}}^{\mathrm{\prime }}$ of all operators $R\in \mathcal{B}\left(E\right)$ such that $SR=RS$ for all $S\in \mathcal{S}$. A subspace $M\subseteq E$ is said to be hyperinvariant under an operator $T\in \mathcal{B}\left(E\right)$ provided that M is invariant under ${\left\{T\right\}}^{\mathrm{\prime }}$.

The invariant subspace problem is the question of whether every operator in $\mathcal{B}\left(E\right)$ has a nontrivial invariant subspace. This is one of the most important open problems in operator theory.

The origin of this question goes back to 1935, when von Neumann proved the unpublished result that any compact operator on a Hilbert space has a nontrivial invariant subspace. Aronszajn and Smith [8] extended this result in 1954 to general Banach spaces. Bernstein and Robinson [9] used nonstandard analysis to prove in 1966 that every polynomially compact operator on a Hilbert space has a nontrivial invariant subspace. Halmos [10] obtained a proof of the same result using classical methods.

Lomonosov [2] proved in 1973 that any nonscalar operator on a Banach space that commutes with a nonzero compact operator has a nontrivial hyperinvariant subspace. The result of Lomonosov came into the scene like a lightning bolt in a clear sky, generalizing all the previously known results and introducing the use of the Schauder fixed point theorem as a new technique to produce invariant subspaces.

Enflo [11] constructed in 1976 the first example of an operator on a Banach space without nontrivial invariant subspaces. The example circulated in a preprint form and it did not appear published until 1987, when it was recognized as correct work [12]. In the meantime, Beauzamy [13] simplified the technique, and further examples were given by Read [14, 15].

Very recently, Argyros and Haydon [16] constructed an example of an infinite dimensional, separable Banach space such that every continuous operator is the sum of a compact operator and a scalar operator, so that every operator on it has a nontrivial invariant subspace.

However, after so many decades, the question about the existence of invariant subspaces for operators on Hilbert space is still an open problem.

## 2 Invariant subspaces for algebras containing compact operators

We start with a fixed point theorem that is the key to the main result in this section. The use of this result is one of the main ideas in the technique of Lomonosov. We shall denote by $\overline{conv}\left(S\right)$ the closed convex hull of a subset $S\subseteq E$.

Proposition 2.1 [[17], Proposition 1]

Let E be a Banach space, let $C\subseteq E$ be a closed convex set, and let $\mathrm{\Phi }:C\to E$ be a continuous mapping such that $\mathrm{\Phi }\left(C\right)$ is a relatively compact subset of C. Then there is a point ${x}_{0}\in C$ such that $\mathrm{\Phi }\left({x}_{0}\right)={x}_{0}$.

Proof Let Q denote the closure of $\mathrm{\Phi }\left(C\right)$. It follows from a theorem of Mazur that $\overline{conv}\left(Q\right)$ is a compact, convex subset of E, and since C is closed and convex, we have $\overline{conv}\left(Q\right)\subseteq C$. Since $\mathrm{\Phi }\left(C\right)\subseteq Q$, we have $\mathrm{\Phi }\left(\overline{conv}\left(Q\right)\right)\subseteq Q\subseteq \overline{conv}\left(Q\right)$, and now the result follows from the Schauder fixed point theorem. □

Theorem 2.2 [[17], Theorem 2]

Let $\mathcal{R}\subseteq \mathcal{B}\left(E\right)$ be a transitive algebra and let $K\in \mathcal{B}\left(E\right)$ be a nonzero compact operator. Then there is an operator $R\in \mathcal{R}$ and there is a vector ${x}_{1}\in E$ such that $RK{x}_{1}={x}_{1}$.

Proof We may assume, without loss of generality, that $\parallel K\parallel =1$. Choose an ${x}_{0}\in E$ such that $\parallel K{x}_{0}\parallel >1$, so that $\parallel {x}_{0}\parallel >1$. Consider the closed ball $B=\left\{x\in E:\parallel x-{x}_{0}\parallel \le 1\right\}$. Then, for each $R\in \mathcal{R}$, consider the open set ${G}_{R}=\left\{y\in E:\parallel Ry-{x}_{0}\parallel <1\right\}$. Since is a transitive algebra, we have

$\bigcup _{R\in \mathcal{R}}{G}_{R}=E\mathrm{\setminus }\left\{0\right\}.$

Since K is a compact operator, $\overline{KB}$ is a compact subset of E, and since $\parallel K\parallel =1$ and $\parallel K{x}_{0}\parallel >1$, we have $0\notin \overline{KB}$. Thus, the family $\left\{{G}_{R}:R\in \mathcal{R}\right\}$ is an open cover of $\overline{KB}$. Hence, there exist finitely many operators ${R}_{1},\dots ,{R}_{n}\in \mathcal{R}$ such that

$\overline{KB}\subseteq \bigcup _{i=1}^{n}{G}_{{R}_{i}}.$

Next, for each $y\in \overline{KB}$ and $i=1,\dots ,n$ we define ${\alpha }_{i}\left(y\right)=max\left\{0,1-\parallel {R}_{i}y-{x}_{0}\parallel \right\}$. Then $0\le {\alpha }_{i}\left(y\right)\le 1$, and for each $y\in \overline{KB}$, there is an $i=1,\dots ,n$ such that $y\in {G}_{{R}_{i}}$, so that ${\alpha }_{i}\left(y\right)>0$. Thus, ${\sum }_{i=1}^{n}{\alpha }_{i}\left(y\right)>0$ for each $y\in \overline{KB}$, and we may define

${\beta }_{i}\left(y\right)=\frac{{\alpha }_{i}\left(y\right)}{{\sum }_{j=1}^{n}{\alpha }_{j}\left(y\right)}$

for $i=1,\dots ,n$ and $y\in \overline{KB}$. Now, each ${\beta }_{i}$ is a continuous function from $\overline{KB}$ into . Hence, we may define a continuous mapping $\mathrm{\Phi }:B\to E$ by the expression

$\mathrm{\Phi }\left(x\right)=\sum _{i=1}^{n}{\beta }_{i}\left(Kx\right){R}_{i}Kx.$

We claim that $\mathrm{\Phi }\left(B\right)\subseteq B$. Indeed, for each $x\in B$, we have ${\sum }_{i=1}^{n}{\beta }_{i}\left(Kx\right)=1$ so that

$\parallel \mathrm{\Phi }\left(x\right)-{x}_{0}\parallel =\parallel \sum _{i=1}^{n}{\beta }_{i}\left(Kx\right)\left({R}_{i}Kx-{x}_{0}\right)\parallel \le \sum _{i=1}^{n}{\beta }_{i}\left(Kx\right)\parallel {R}_{i}Kx-{x}_{0}\parallel .$

If $\parallel {R}_{i}Kx-{x}_{0}\parallel >1$, then ${\alpha }_{i}\left(Kx\right)=0$ and therefore ${\beta }_{i}\left(Kx\right)=0$. Hence,

$\parallel \mathrm{\Phi }\left(x\right)-{x}_{0}\parallel \le \sum _{i=1}^{n}{\beta }_{i}\left(Kx\right)=1,$

and this completes the proof of our claim. Finally, each operator ${R}_{i}K$ is compact so that each ${R}_{i}KB$ is relatively compact, and it follows from an earlier mentioned theorem of Mazur that $Q=\overline{conv}{\bigcup }_{i=1}^{n}{R}_{i}KB$ is compact. Since $\mathrm{\Phi }\left(B\right)\subseteq Q$, the set $\mathrm{\Phi }\left(B\right)$ is a relatively compact subset of B. Now, we apply Proposition 2.1 to find a vector ${x}_{1}\in B$ such that $\mathrm{\Phi }\left({x}_{1}\right)={x}_{1}$. Since $0\notin B$, we have ${x}_{1}\ne 0$. Then we consider the operator defined by

$Rx=\sum _{i=1}^{n}{\beta }_{i}\left(K{x}_{1}\right){R}_{i}x,$

and we conclude that $R\in \mathcal{R}$ and $RK{x}_{1}={x}_{1}$, as we wanted. □

Corollary 2.3 [2], [[17], Theorem 3]

Every nonscalar operator that commutes with a nonzero compact operator has a nontrivial, hyperinvariant subspace.

Proof Let $T\in \mathcal{B}\left(E\right)$ be a nonscalar operator and suppose that T commutes with a nonzero compact operator K. We must show that the commutant ${\left\{T\right\}}^{\mathrm{\prime }}$ is nontransitive. Suppose, on the contrary, that ${\left\{T\right\}}^{\mathrm{\prime }}$ is transitive. We can apply Theorem 2.2 to find an operator $R\in {\left\{T\right\}}^{\mathrm{\prime }}$ such that $\lambda =1$ is an eigenvalue of the compact operator RK with an associated finite dimensional eigenspace $F=ker\left(RK-I\right)$. Since T commutes with RK, we observe that T maps F into itself, and therefore, T must have an eigenvalue. Since T is nonscalar, the corresponding eigenspace M cannot be the whole E, and it is invariant under ${\left\{T\right\}}^{\mathrm{\prime }}$. The contradiction has arrived. □

## 3 Invariant subspaces for localizing algebras

In this section we use the following fixed point theorem of Ky Fan [18]. Recall that if Ω is a topological space and $\mathrm{\Phi }:\mathrm{\Omega }\to \mathcal{P}\left(\mathrm{\Omega }\right)$ is a point to set map from Ω to the power set of Ω, then Φ is said to be upper semicontinuous if for every ${x}_{0}\in \mathrm{\Omega }$ and every open set $U\subseteq \mathrm{\Omega }$ such that $\mathrm{\Phi }\left({x}_{0}\right)\subseteq U$, there is a neighborhood V of ${x}_{0}$ such that $\mathrm{\Phi }\left(x\right)\subseteq U$ for every $x\in V$. In terms of convergence of nets, this definition is equivalent to saying that for every $x\in \mathrm{\Omega }$, for every net $\left({x}_{\alpha }\right)$ with ${x}_{\alpha }\to x$, and for every ${y}_{\alpha }\in \mathrm{\Phi }\left({x}_{\alpha }\right)$ such that the net $\left({y}_{\alpha }\right)$ converges to some $y\in \mathrm{\Omega }$, we have $y\in \mathrm{\Phi }\left(x\right)$.

Theorem 3.1 (Ky Fan fixed point theorem [18])

Let C be a compact convex subset of a locally convex space, and let $\mathrm{\Phi }:C\to \mathcal{P}\left(C\right)$ be an upper semicontinuous mapping such that $\mathrm{\Phi }\left(x\right)$ is a nonempty, closed convex set for every $x\in C$. Then there is an ${x}_{0}\in C$ such that ${x}_{0}\in \mathrm{\Phi }\left({x}_{0}\right)$.

A subalgebra $\mathcal{R}\subseteq \mathcal{B}\left(H\right)$ is said to be strongly compact if its unit ball is precompact in the strong operator topology. An important example of a strongly compact algebra is the commutant of a compact operator with a dense range. We shall denote by $ball\left(\mathcal{R}\right)$ the unit ball of .

This notion was introduced by Lomonosov [19] as a means to prove the existence of invariant subspaces for essentially normal operators on Hilbert spaces. Recall that an operator T on a Hilbert space is said to be essentially normal if ${T}^{\ast }T-T{T}^{\ast }$ is a compact operator. Lomonosov showed that if an essentially normal operator T has the property that both its commutant ${\left\{T\right\}}^{\mathrm{\prime }}$ and the commutant of its adjoint ${\left\{{T}^{\ast }\right\}}^{\mathrm{\prime }}$ fail to be strongly compact, then T has a nontrivial invariant subspace.

Thus, in order to solve the invariant subspace problem for essentially normal operators, it suffices to consider only operators with a strongly compact commutant.

Lomonosov, Radjavi, and Troitsky [3] obtained a result about the existence of invariant subspaces for an operator with a strongly compact commutant under the additional assumption that the commutant of the adjoint is a localizing algebra.

A subalgebra $\mathcal{R}\subseteq \mathcal{B}\left(E\right)$ is said to be localizing provided that there is a closed ball $B\subseteq E$ such that $0\notin B$ and such that for every sequence $\left({x}_{n}\right)$ in B there is a subsequence $\left({x}_{{n}_{j}}\right)$, and a sequence of operators $\left({R}_{j}\right)$ in such that $\parallel {R}_{j}\parallel \le 1$ and $\left({R}_{j}{x}_{{n}_{j}}\right)$ converges in norm to some nonzero vector. An important example of a localizing algebra is any algebra containing a nonzero compact operator.

Proposition 3.2 [[3], Proof of Theorem 2.3]

Let $\mathcal{R}\subseteq \mathcal{B}\left(E\right)$ be a transitive localizing algebra, let $B\subseteq E$ be a closed ball as above, and let $T\in {\mathcal{R}}^{\mathrm{\prime }}$ be a nonzero operator. Then there exists an $r>0$ such that for every $x\in B$ we have $rball\left(\mathcal{R}\right)\left(Tx\right)\cap B\ne \mathrm{\varnothing }$.

Proof First, T is one-to-one because is transitive and kerT is invariant under . If this is not so, then for every $n\ge 1$, there is a vector ${x}_{n}\in B$ such that $\parallel R\parallel \ge n$, whenever $R\in \mathcal{R}$ and $RT{x}_{n}\in B$. Since is localizing, there is a subsequence $\left({x}_{{n}_{j}}\right)$ and a sequence $\left({R}_{j}\right)$ in such that $\parallel {R}_{j}\parallel \le 1$ and $\left({R}_{j}{x}_{{n}_{j}}\right)$ converges in norm to some nonzero vector $x\in X$. We have $T{R}_{j}={R}_{j}T$ for all $j\ge 1$, so that $\left({R}_{j}T{x}_{{n}_{j}}\right)$ converges to Tx in norm. Now $Tx\ne 0$ because T is injective and $x\ne 0$. Since is transitive, there is an operator $R\in \mathcal{R}$ such that $RTx\in intB$. It follows that there is a ${j}_{0}\ge 1$ such that $R{R}_{j}T{x}_{{n}_{j}}\in intB$ for every $j\ge {j}_{0}$. Since $R{R}_{j}\in \mathcal{R}$, the choice of the sequence $\left({x}_{n}\right)$ implies that $\parallel R{R}_{j}\parallel \ge {n}_{j}$ for every $j\ge {j}_{0}$, and this is a contradiction because $\parallel R{R}_{j}\parallel \le \parallel R\parallel$ for every $j\ge 1$. □

If E is a Banach space, then ${E}^{\ast }$ denotes its dual space. If $\mathcal{R}\subseteq \mathcal{B}\left(E\right)$ is a subalgebra, then ${\mathcal{R}}^{\ast }$ denotes the subalgebra of $\mathcal{B}\left({E}^{\ast }\right)$ of the adjoints of the elements of , that is, ${\mathcal{R}}^{\ast }=\left\{{R}^{\ast }:R\in \mathcal{R}\right\}$.

Theorem 3.3 [[3], Theorem 2.3]

Let E be a complex Banach space, let $\mathcal{R}\subseteq \mathcal{B}\left(E\right)$ be a strongly compact subalgebra such that ${\mathcal{R}}^{\ast }$ is a transitive localizing algebra and it is closed in the weak- operator topology. If $T\in {\mathcal{R}}^{\mathrm{\prime }}$ is a nonzero operator, then there is an operator $R\in \mathcal{R}$ and there is a nonzero vector ${x}^{\ast }\in {E}^{\ast }$ such that ${R}^{\ast }{T}^{\ast }{x}^{\ast }={x}^{\ast }$. Moreover, the operator ${T}^{\ast }$ has a nontrivial invariant subspace.

Proof We shall apply Proposition 3.2 to the algebra ${\mathcal{R}}^{\ast }$. Let ${B}^{\ast }\subseteq {E}^{\ast }$ be a closed ball as in the definition of a localizing algebra, let $r>0$ be a positive number as in Proposition 3.2, and define a multivalued map $\mathrm{\Phi }:{B}^{\ast }\to \mathcal{P}\left({B}^{\ast }\right)$ by the expression

$\mathrm{\Phi }\left({x}^{\ast }\right)=rball\left({\mathcal{R}}^{\ast }\right)\left({T}^{\ast }{x}^{\ast }\right)\cap {B}^{\ast }.$

Then, $\mathrm{\Phi }\left({x}^{\ast }\right)$ is a nonempty, convex subset of ${B}^{\ast }$. Also, $\mathrm{\Phi }\left({x}^{\ast }\right)$ is weak- closed because $ball\left({\mathcal{R}}^{\ast }\right)\left({T}^{\ast }{x}^{\ast }\right)$ is weak- compact as the image of $ball\left({\mathcal{R}}^{\ast }\right)$ under the map ${R}^{\ast }\to {R}^{\ast }{T}^{\ast }{x}^{\ast }$, which is continuous from $\mathcal{B}\left({E}^{\ast }\right)$ with the weak- operator topology into ${E}^{\ast }$ with the weak- topology, and $ball\left({\mathcal{R}}^{\ast }\right)$ is compact in the weak- operator topology.

We claim that Φ is upper semicontinuous for the weak- topology. Indeed, let ${x}^{\ast },{y}^{\ast }\in {B}^{\ast }$, and let $\left({x}_{\alpha }^{\ast }\right)$ and $\left({y}_{\alpha }^{\ast }\right)$ be two nets in ${B}^{\ast }$ with ${x}_{\alpha }^{\ast }\to {x}^{\ast }$, ${y}_{\alpha }^{\ast }\to {y}^{\ast }$ in the weak- topology and such that ${y}_{\alpha }^{\ast }\in \mathrm{\Phi }\left({x}_{\alpha }^{\ast }\right)$. We must show that ${y}^{\ast }\in \mathrm{\Phi }\left({x}^{\ast }\right)$. Since ${y}_{\alpha }^{\ast }\in \mathrm{\Phi }\left({x}_{\alpha }^{\ast }\right)$, there is an ${R}_{\alpha }^{\ast }\in ball\left({\mathcal{R}}^{\ast }\right)$ such that ${y}_{\alpha }^{\ast }=r{R}_{\alpha }^{\ast }{T}^{\ast }{x}_{\alpha }^{\ast }$. Since $ball\left(\mathcal{R}\right)$ is precompact in the strong operator topology, there exists a subnet $\left({R}_{{\alpha }_{\beta }}\right)$ that converges in the strong operator topology to some $R\in \mathcal{B}\left(E\right)$. Thus, ${R}_{{\alpha }_{\beta }}^{\ast }\to {R}^{\ast }$ in the weak- operator topology. Notice that $ball\left({\mathcal{R}}^{\ast }\right)$ is compact in this topology because $ball\left(\mathcal{B}\left({E}^{\ast }\right)\right)$ is compact in this topology and ${\mathcal{R}}^{\ast }$ is closed in this topology. It follows that ${R}^{\ast }\in ball\left({\mathcal{R}}^{\ast }\right)$. Let $x\in E$ and notice that $\parallel T{R}_{{\alpha }_{\beta }}x-TRx\parallel \to 0$. Then

$\begin{array}{rcl}〈x,{y}_{{\alpha }_{\beta }}^{\ast }〉& =& 〈x,r{R}_{{\alpha }_{\beta }}^{\ast }{T}^{\ast }{x}_{{\alpha }_{\beta }}^{\ast }〉\\ =& r〈T{R}_{{\alpha }_{\beta }}x,{x}_{{\alpha }_{\beta }}^{\ast }〉\\ =& r〈T{R}_{{\alpha }_{\beta }}x-TRx,{x}_{{\alpha }_{\beta }}^{\ast }〉+r〈TRx,{x}_{{\alpha }_{\beta }}^{\ast }〉.\end{array}$

We have $〈T{R}_{{\alpha }_{\beta }}x-TRx,{x}_{{\alpha }_{\beta }}^{\ast }〉\to 0$ and $〈TRx,{x}_{{\alpha }_{\beta }}^{\ast }〉\to 〈TRx,{x}^{\ast }〉=〈x,{R}^{\ast }{T}^{\ast }{x}^{\ast }〉$, so that

$〈x,{y}_{{\alpha }_{\beta }}^{\ast }〉\to 〈x,r{R}^{\ast }{T}^{\ast }{x}^{\ast }〉.$

Since $x\in E$ is arbitrary, ${y}_{{\alpha }_{\beta }}^{\ast }\to r{R}^{\ast }{T}^{\ast }{x}^{\ast }$ in the weak- topology, and it follows that ${y}^{\ast }=r{R}^{\ast }{T}^{\ast }{x}^{\ast }$. This shows that ${y}^{\ast }\in \mathrm{\Phi }\left({x}^{\ast }\right)$, and the proof of our claim is complete.

Since the map Φ is upper semicontinuous and ${B}^{\ast }$ is compact in the weak- topology, it follows from the Ky Fan fixed point theorem that there is a vector ${x}^{\ast }\in {B}^{\ast }$ such that ${x}^{\ast }\in \mathrm{\Phi }\left({x}^{\ast }\right)$; that is, there is an operator $R\in ball\left(\mathcal{R}\right)$ such that ${x}^{\ast }=r{R}^{\ast }{T}^{\ast }{x}^{\ast }$.

Finally, consider the closed subspace defined as $M=ker\left({R}^{\ast }{T}^{\ast }-I\right)$. Notice that M is invariant under ${T}^{\ast }$ and $M\ne \left\{0\right\}$. If ${T}^{\ast }$ is not invertible then $M\ne E$ and we are done. If ${T}^{\ast }$ is invertible, pick any $\lambda \in \sigma \left({T}^{\ast }\right)$ and put $S=\lambda -{T}^{\ast }$. Then S is not invertible and the preceding argument applied to S shows that S has a nontrivial invariant subspace. It is clear that such subspace is also invariant under ${T}^{\ast }$. □

Corollary 3.4 [[3], Corollary 2.4]

Let $T\in \mathcal{B}\left(E\right)$ be an operator such that ${\left\{T\right\}}^{\mathrm{\prime }}$ is a strongly compact algebra and ${\left\{{T}^{\ast }\right\}}^{\mathrm{\prime }}$ is a localizing algebra. Then ${T}^{\ast }$ has a nontrivial invariant subspace.

Proof If ${T}^{\ast }$ has a hyperinvariant subspace then there is nothing to prove, and otherwise ${\left\{{T}^{\ast }\right\}}^{\mathrm{\prime }}$ is a transitive algebra so that Theorem 3.3 applies. □

Notice that the assumptions of Corollary 3.4 are met whenever T is a compact operator with a dense range.

## 4 An infinite dimensional version of Burnside’s theorem

Burnside’s classical theorem is the assertion that for a finite dimensional linear space F, the only transitive subalgebra of $\mathcal{B}\left(F\right)$ is the whole algebra $\mathcal{B}\left(F\right)$. Lomonosov [5] obtained a generalization of Burnside’s theorem to infinite dimensional Banach spaces. Scott Brown [6] proved the same result independently for the special case of a Hilbert space and a commutative algebra. Lindström and Schlüchtermann [20] provided a relatively short proof of the Lomonosov result in full generality. In this section we present a proof of the Scott Brown result that is based on the Schauder fixed point theorem.

Let H be a complex, infinite dimensional, and separable Hilbert space. Let $T\in \mathcal{B}\left(H\right)$, and let ${\parallel T\parallel }_{e}$ denote the essential norm of T, that is, the distance from T to the space of compact operators.

Theorem 4.1 [[6], Theorem 1.1]

Let be a commutative subalgebra of $\mathcal{B}\left(H\right)$. Then there exist nonzero vectors $x,y\in H$ such that for any $R\in \mathcal{R}$ we have $|〈Rx,y〉|\le {\parallel R\parallel }_{e}$.

Proof Consider the set $\mathcal{E}=\left\{R\in \mathcal{R}:{\parallel R\parallel }_{e}\le 1/16\right\}$. We claim that there is some $x\in H\mathrm{\setminus }\left\{0\right\}$ such that the set $\mathcal{E}x$ is not dense in H. The result then follows easily because in that case there is some $y\in H\mathrm{\setminus }\left\{0\right\}$ such that $|〈Rx,y〉|\le 1$ for all $R\in \mathcal{E}$. Now, for the proof of our claim, we proceed by contradiction. Suppose that the set $\mathcal{E}x$ is dense in H for every $x\in H\mathrm{\setminus }\left\{0\right\}$. Choose ${x}_{0}\in H$ with $\parallel {x}_{0}\parallel =2$ and consider the closed ball $B=\left\{x\in H:\parallel x-{x}_{0}\parallel \le 1\right\}$. Then, for every vector $x\in B$, there is an operator ${R}_{x}\in \mathcal{E}$ such that $\parallel {R}_{x}x-{x}_{0}\parallel <1/2$. Next, there is a bounded operator ${T}_{x}$ and a compact operator ${K}_{x}$ such that ${R}_{x}={T}_{x}+{K}_{x}$ and $\parallel {T}_{x}\parallel \le 1/8$. Since ${K}_{x}$ is a compact operator, it is weak-to-norm continuous on bounded sets so that there exists an open neighborhood of x in the weak topology, say ${V}_{x}\subseteq H$, such that $\parallel {K}_{x}y-{K}_{x}x\parallel <1/4$ for all $y\in {V}_{x}\cap B$. Then consider the set ${U}_{x}={V}_{x}\cap B$ and notice that ${U}_{x}$ is an open neighborhood of x in the weak topology relative to B. Moreover, for $y\in {U}_{x}$ we have

$\parallel {R}_{x}y-{R}_{x}x\parallel \le \parallel {T}_{x}y-{T}_{x}x\parallel +\parallel {K}_{x}y-{K}_{x}x\parallel <2\cdot \frac{1}{8}+\frac{1}{4}=\frac{1}{2},$

and therefore $\parallel {R}_{x}y-{x}_{0}\parallel <1$. Hence, ${R}_{x}{U}_{x}\subseteq B$. Since B is compact in the weak topology, there exist finitely many vectors ${x}_{1},\dots {x}_{n}\in B$ such that

$B\subseteq \bigcup _{j=1}^{n}{U}_{{x}_{j}}.$

Choose some weakly continuous functions ${f}_{1},\dots ,{f}_{n}$ on B such that , $0\le {f}_{j}\left(x\right)\le 1$, and

Define a weakly continuous mapping $\mathrm{\Phi }:B\to B$ by the expression

and apply the Schauder fixed point theorem to find a vector ${y}_{0}\in B$ such that $\mathrm{\Phi }\left({y}_{0}\right)={y}_{0}$. Finally, consider the operator $R\in \mathcal{R}$ defined by the expression

$R=\sum _{j=1}^{n}{f}_{j}\left({y}_{0}\right){R}_{{x}_{j}}.$

Hence, $R{y}_{0}={y}_{0}$. Notice that $R\ne I$ because ${\parallel R\parallel }_{e}\le 1/8$. Then, the eigenspace $M=\left\{x\in H:Rx=x\right\}$ is a closed nontrivial invariant subspace for the algebra . Thus, any vector $x\in M$ has the property that the set $\mathcal{E}x$ is not dense in H. The contradiction has arrived. □

## 5 Invariant subspaces for operators on the Krein space

Let ${H}_{1}$, ${H}_{2}$ be two Hilbert spaces and consider the orthogonal direct sum $H={H}_{1}\oplus {H}_{2}$. Let ${P}_{1}$, ${P}_{2}$ denote the orthogonal projections from H onto ${H}_{1}$, ${H}_{2}$, respectively. Consider the operator $J:={P}_{1}-{P}_{2}$. The Krein space is the space H provided with the indefinite product

$\left[x,y\right]:=〈Jx,y〉,\phantom{\rule{1em}{0ex}}x,y\in H.$

Notice that J is a selfadjoint involution, that is, ${J}^{\ast }=J$ and ${J}^{2}=I$. The operator J is sometimes called the fundamental symmetry of the Krein space.

A vector $x\in H$ is said to be nonnegative provided that $\left[x,x\right]\ge 0$, and a subspace $M\subseteq H$ is said to be nonnegative provided that $\left[x,x\right]\ge 0$ for all $x\in M$.

Every operator $T\in \mathcal{B}\left(H\right)$ has a matrix representation

$T=\left[\begin{array}{cc}{T}_{11}& {T}_{12}\\ {T}_{21}& {T}_{22}\end{array}\right]$

with respect to the decomposition $H={H}_{1}\oplus {H}_{2}$.

There is a natural, one-to-one and onto correspondence between the maximal nonnegative invariant subspaces M of an operator $T\in \mathcal{B}\left(H\right)$ and the contractive solutions $X\in \mathcal{B}\left({H}_{1},{H}_{2}\right)$ of the so-called operator Riccati equation

$X{T}_{12}X+X{T}_{11}-{T}_{22}X-{T}_{21}=0.$

The correspondence $X↔M$ is given by $M=\left\{{x}_{1}\oplus X{x}_{1}:{x}_{1}\in {H}_{1}\right\}$, where $\parallel X\parallel \le 1$. The operator T is usually called the Hamiltonian operator of the operator Ricatti equation.

An operator $T\in \mathcal{B}\left(H\right)$ is said to be J-selfadjoint provided that $\left[Tx,y\right]=\left[x,Ty\right]$ for every $x,y\in H$. This is equivalent to saying that $JT={T}^{\ast }J$, or in other words, ${T}_{11}^{\ast }={T}_{11}$, ${T}_{22}^{\ast }={T}_{22}$, and ${T}_{12}^{\ast }=-{T}_{21}$.

A classical theorem of Krein is the assertion that if the Hamiltonian operator T is J-selfadjoint and the corner operator ${T}_{12}$ is compact, then there exists a maximal nonnegative invariant subspace for T.

Albeverio, Makarov, and Motovilov [7] addressed the question of the existence and uniqueness of contractive solutions to the operator Riccati equation under the condition that the diagonal entries in the Hamiltonian operator have disjoint spectra, that is, $\sigma \left({T}_{11}\right)\cap \sigma \left({T}_{22}\right)=\mathrm{\varnothing }$. They proved the following

Theorem 5.1 [[7], Theorem 3.6 and Lemma 3.11]

There is some universal constant $c>0$ such that whenever the corner operator ${T}_{12}$ satisfies the condition

$\parallel {T}_{12}\parallel

there is a unique solution X to the operator Riccati equation with $\parallel X\parallel \le 1$.

An earlier result in this direction was given by Motovilov [[21], Corollary 1] with the stronger assumption that the corner operator ${T}_{12}$ is Hilbert-Schmidt. Adamjan, Langer, and Tretter [22] extended the technique to the case that the Hamiltonian operator is not J-selfadjoint. Kostrykin, Makarov, and Motovilov [23] adopted the assumption that $\sigma \left({T}_{11}\right)$ lies in a gap of $\sigma \left({T}_{22}\right)$ and they showed that the best constant, in that context, is $c=\sqrt{2}$.

We present a proof of Theorem 5.2 that is based on the Banach fixed point theorem. This method can be found in the paper of Albeverio, Motovilov, and Shkalikov [[24], Theorem 4.1]. A basic tool is the bounded linear operator R defined for $X\in \mathcal{B}\left({H}_{1},{H}_{2}\right)$ by the expression

$R\left(X\right):={T}_{22}X-X{T}_{11}.$

It follows from the Rosenblum theorem that the map R is invertible. The main result is the following

Theorem 5.2 [[24], Theorem 4.1]

If the operators ${T}_{11}$, ${T}_{22}$ have disjoint spectra and the corner operator ${T}_{12}$ satisfies the estimate

$\parallel {T}_{12}\parallel <\frac{1}{2\parallel {R}^{-1}\parallel },$

then there is a unique solution X to the operator Riccati equation with $\parallel X\parallel \le 1$.

The following upper bound on the norm of the inverse ${R}^{-1}$ can be found in the work of Albeverio, Makarov, and Motovilov [[7], Theorem 2.7]. See also the paper by Bhatia and Rosenthal [[4], p.15] for this interesting result and other related issues.

Theorem 5.3 [[7], Theorem 2.7]

If the operators ${T}_{11}$, ${T}_{22}$ have disjoint spectra, then

$\parallel {R}^{-1}\parallel \le \frac{\pi }{2}\cdot \frac{1}{dist\left[\sigma \left({T}_{11}\right),\sigma \left({T}_{22}\right)\right]}.$

Notice that Theorem 5.1 becomes a corollary of Theorem 5.2 and Theorem 5.3 with the constant $c=1/\pi$.

Proof of Theorem 5.2 Consider the quadratic map Q defined for $X\in \mathcal{B}\left({H}_{1},{H}_{2}\right)$ by the expression

$Q\left(X\right):=X{T}_{12}X-{T}_{21}.$

It is clear that the operator Riccati equation can be expressed as

$Q\left(X\right)-R\left(X\right)=0,$

or equivalently, $X={R}^{-1}\left(Q\left(X\right)\right)$. Thus, the solutions of the operator Riccati equation are the fixed points of the map $S:={R}^{-1}\circ Q$. Now, let us check that the map S takes the unit ball of $\mathcal{B}\left({H}_{1},{H}_{2}\right)$ into itself. Indeed, if $\parallel X\parallel \le 1$, then

$\begin{array}{rcl}\parallel S\left(X\right)\parallel & =& \parallel {R}^{-1}\left(Q\left(X\right)\right)\parallel \le \parallel {R}^{-1}\parallel \cdot \parallel Q\left(X\right)\parallel \\ \le & \parallel {R}^{-1}\parallel \cdot \left(\parallel {T}_{12}\parallel \cdot {\parallel X\parallel }^{2}+\parallel {T}_{21}\parallel \right)\\ \le & \parallel {R}^{-1}\parallel \cdot \left(\parallel {T}_{12}\parallel +\parallel {T}_{21}\parallel \right)=2\parallel {R}^{-1}\parallel \cdot \parallel {T}_{12}\parallel <1.\end{array}$

Also, the map S is contractive, for if $\parallel X\parallel ,\parallel Y\parallel \le 1$, then

$\begin{array}{rcl}\parallel Q\left(X\right)-Q\left(Y\right)\parallel & =& \parallel X{T}_{12}X-Y{T}_{12}Y\parallel \\ \le & \parallel X{T}_{12}X-X{T}_{12}Y\parallel +\parallel X{T}_{12}Y-Y{T}_{12}Y\parallel \\ \le & \left(\parallel X\parallel +\parallel Y\parallel \right)\cdot \parallel {T}_{12}\parallel \cdot \parallel X-Y\parallel \le 2\parallel {T}_{12}\parallel \cdot \parallel X-Y\parallel ,\end{array}$

and from this inequality it follows that

$\begin{array}{rcl}\parallel S\left(X\right)-S\left(Y\right)\parallel & =& \parallel {R}^{-1}\left(Q\left(X\right)-Q\left(Y\right)\right)\parallel \\ \le & \parallel {R}^{-1}\parallel \cdot \parallel Q\left(X\right)-Q\left(Y\right)\parallel \le 2\parallel {R}^{-1}\parallel \cdot \parallel {T}_{12}\parallel \cdot \parallel X-Y\parallel \end{array}$

so that the map S satisfies a Lipschitz condition with a Lipschitz constant $2\parallel {R}^{-1}\parallel \cdot \parallel {T}_{12}\parallel <1$. The result now follows at once as a consequence of the Banach fixed point theorem. □

## References

1. Yadav BS: The present state and heritages of the invariant subspace problem. Milan J. Math. 2005, 73: 289–316. 10.1007/s00032-005-0048-7

2. Lomonosov VI: Invariant subspaces of the family of operators that commute with a completely continuous operator. Funkc. Anal. Prilozh. 1973, 7(3):55–56.

3. Lomonosov VI, Radjavi H, Troitsky VG: Sesquitransitive and localizing operator algebras. Integral Equ. Oper. Theory 2008, 60(3):405–418. 10.1007/s00020-008-1560-2

4. Androulakis G: A new method for constructing invariant subspaces. J. Math. Anal. Appl. 2007, 333(2):1254–1263. 10.1016/j.jmaa.2006.12.023

5. Lomonosov VI: An extension of Burnside’s theorem to infinite-dimensional spaces. Isr. J. Math. 1991, 75(2–3):329–339. 10.1007/BF02776031

6. Brown SW: Lomonosov’s theorem and essentially normal operators. N.Z. J. Math. 1994, 23: 11–18.

7. Albeverio S, Makarov KA, Motovilov AK: Graph subspaces and the spectral shift function. Can. J. Math. 2003, 55(3):449–503. 10.4153/CJM-2003-020-7

8. Aronszajn N, Smith KT: Invariant subspaces of completely continuous operators. Ann. of Math. (2) 1954, 60: 345–350. 10.2307/1969637

9. Bernstein AR, Robinson A: Solution of an invariant subspace problem of K. T. Smith and P. R. Halmos. Pac. J. Math. 1966, 16: 421–431. 10.2140/pjm.1966.16.421

10. Halmos PR: Invariant subspaces of polynomially compact operators. Pac. J. Math. 1966, 16: 433–437. 10.2140/pjm.1966.16.433

11. Enflo P: On the invariant subspace problem in Banach spaces. In Séminaire Maurey-Schwartz (1975–1976) Espaces \$L^{p}\$Lp, applications radonifiantes et géométrie des espaces de Banach, Exp. Nos. 14–15. Centre Math., École Polytech., Palaiseau; 1976.

12. Enflo P: On the invariant subspace problem for Banach spaces. Acta Math. 1987, 158(3–4):213–313.

13. Beauzamy B: Un opérateur sans sous-espace invariant: simplification de l’exemple de P. Enflo. Integral Equ. Oper. Theory 1985, 8(3):314–384. 10.1007/BF01202903

14. Read CJ: A solution to the invariant subspace problem. Bull. Lond. Math. Soc. 1984, 16(4):337–401. 10.1112/blms/16.4.337

15. Read CJ: A solution to the invariant subspace problem on the space ${l}_{1}$. Bull. Lond. Math. Soc. 1985, 17(4):305–317. 10.1112/blms/17.4.305

16. Argyros SA, Haydon RG: A hereditarily indecomposable ${\mathcal{L}}_{\mathrm{\infty }}$-space that solves the scalar-plus-compact problem. Acta Math. 2011, 206: 1–54. 10.1007/s11511-011-0058-y

17. Pearcy C, Shields AL: A survey of the Lomonosov technique in the theory of invariant subspaces. Math. Surveys 13. In Topics in operator theory. Am. Math. Soc., Providence; 1974:219–229.

18. Fan K: Fixed-point and minimax theorems in locally convex topological linear spaces. Proc. Natl. Acad. Sci. USA 1952, 38: 121–126. 10.1073/pnas.38.2.121

19. Lomonosov VI: A construction of an intertwining operator. Funkc. Anal. Prilozh. 1980, 14: 67–68.

20. Lindström M, Schlüchtermann G: Lomonosov’s techniques and Burnside’s theorem. Can. Math. Bull. 2000, 43: 87–89. 10.4153/CMB-2000-013-8

21. Motovilov AK: The removal of an energy dependence from the interaction in two-body systems. J. Math. Phys. 1991, 32(12):3509–3518. 10.1063/1.529466

22. Adamjan V, Langer H, Tretter C: Existence and uniqueness of contractive solutions of some Riccati equations. J. Funct. Anal. 2001, 179(2):448–473. 10.1006/jfan.2000.3680

23. Kostrykin V, Makarov KA, Motovilov AK: Perturbation of spectra and spectral subspaces. Trans. Am. Math. Soc. 2007, 359: 77–89. 10.1090/S0002-9947-06-03930-4

24. Albeverio S, Motovilov AK, Shkalikov AA: Bounds on variation of spectral subspaces under J -self-adjoint perturbations. Integral Equ. Oper. Theory 2009, 64(4):455–486. 10.1007/s00020-009-1702-1

## Acknowledgements

This research was partially supported by Junta de Andalucía under projects FQM-127 and FQM-3737, and by Ministerio de Educación, Cultura y Deporte under projects MTM2012-34847C02-01 and MTM2009-08934.

## Author information

Authors

### Corresponding author

Correspondence to Miguel Lacruz.

### Competing interests

The authors declare that they have no competing interests.

### Authors’ contributions

Both authors contributed equally and significantly in writing this article. Both authors read and approved the final manuscript.

## Rights and permissions

Reprints and permissions

Espínola, R., Lacruz, M. Applications of fixed point theorems in the theory of invariant subspaces. Fixed Point Theory Appl 2012, 197 (2012). https://doi.org/10.1186/1687-1812-2012-197