Strong duality

Primal Problem and Dual Problem

Consider the standard form linear programming problem

\[\begin{aligned} \text{minimize }~~~&\mathbf c'\mathbf x\\ \text{subject to}~~~&\mathbf A\mathbf x=\mathbf b\\&\mathbf x\ge 0 \end{aligned} \]

which we call the primal problem.

Let \(x^{*}\) be an optimal solution, assumed to exist. We introduce a relaxed problem in which the constraint \(\mathbf A\mathbf x=\mathbf b\) is replaced by a penalty \(\mathbf p'(\mathbf b-\mathbf A\mathbf x)\), where \(\mathbf p\) is a price vector of the same dimension as \(\mathbf b\). We then faced with the problem

\[\begin{aligned} \text{minimize }~~~&\mathbf c'\mathbf x+\mathbf p'(\mathbf b-\mathbf A\mathbf x)\\\ \text{subject to}~~~&\mathbf x\ge 0 \end{aligned} \]

Let \(g(\mathbf p)\) be the optimal cost for the relaxed problem. Then we have

\[g(\mathbf p)=\min_{\mathbf x\ge 0}[\mathbf c'\mathbf x+\mathbf p'(\mathbf b-\mathbf A\mathbf x)]\le\mathbf c'\mathbf x^*+\mathbf p'(\mathbf b-\mathbf A\mathbf x^*)=\mathbf c'\mathbf x^* \]

Thus each \(\mathbf p\) leads to a lower bound \(g(\mathbf p)\) for the optimal cost \(\mathbf c'\mathbf x^*\). The problem

\[\begin{aligned} \text{minimize }~~~&g(\mathbf p)\\\ \text{subject to}~~~&\text{no constraints} \end{aligned} \]

can be then interpreted as a search for the tightest possible lower bound of this type, and is known as the dual problem.

Using the definition of \(g(\mathbf p)\), we have

\[g(\mathbf p)=\mathbf p'\mathbf b+\min_{\mathbf x\ge 0}(\mathbf c'-\mathbf p'\mathbf A)\mathbf x=\mathbf p'\mathbf b+\begin{cases}0,&\text{if }\mathbf c'-\mathbf p'\mathbf A\ge \mathbf 0'\\-\infty,&\text{otherwise}\end{cases} \]

Therefore the dual problem is the same as the linear programming problem

\[\begin{aligned} \text{maximize }~~~&\mathbf p'\mathbf b\\ \text{subject to}~~~&\mathbf p'\mathbf A\le \mathbf c' \end{aligned} \]

Weak duality

In the above derivation, we can easily find the following property:

If \(\mathbf x\) is a feasible solution to the primal problem and \(\mathbf p\) is a feasible solution to the dual problem, then

\[\mathbf p'\mathbf b\le \mathbf c'\mathbf x \]

There are two corollaries that are useful later.

(a) If the optimal cost in the primal is \(-\infty\), then the dual problem must be infeasible; if the optimal cost in the dual is \(+\infty\), then the primal problem must be infeasible.

(b) Let \(\mathbf x\) and \(\mathbf p\) be feasible solutions to the primal and the dual, respectively, and suppose \(\mathbf p'\mathbf b=\mathbf c'\mathbf x\). Then \(\mathbf x\) and \(\mathbf p\) are optimal solutions to the primal and the dual, respectively.

Strong duality

The above result is still too weak. The next theorem is the central result on linear programming duality:

If a linear programming problem has an optimal solution, so does its dual, and the respective optimal costs are equal.

The rest of this article is two proofs of this theorem.

Proof 1

Let us assume temporarily that the rows of \(\mathbf A\) are linearly independent and that there exists an optimal solution. Let us apply the simplex method to this problem.

As long as cycling is avoided, e.g., by using the lexicographic pivoting rule, the simplex method terminates with an optimal solution \(\mathbf x\) and an optimal basis \(\mathbf B\). Let \(\mathbf x_B=\mathbf B^{-1}\mathbf b\) be the corresponding vector of basic variables.

When the simplex method terminates, the reduced costs must be nonnegative and we obtain

\[\mathbf c'-\mathbf c'_B\mathbf B^{-1}\mathbf A\ge \mathbf 0' \]

where \(\mathbf c'_B\) is the vector with the costs of the basic variables. Let us define a vector \(\mathbf p\) by letting \(\mathbf p'=\mathbf c_B'B^{-1}\), then we have \(\mathbf p'\mathbf A\le \mathbf c'\), which shows that \(\mathbf p\) is a feasible solution to the dual problem. In addition,

\[\mathbf p'\mathbf b=\mathbf c'_B\mathbf B^{-1}\mathbf b=\mathbf c'_B\mathbf x_B=\mathbf c'\mathbf x \]

It follows that (by the corollary (b)) \(\mathbf p\) is an optimal solution to the dual, and the optimal dual cost is equal to the optimal primal cost.

If we are dealing with a general linear programming problem \(\Pi_1\) that has an optimal solution, we first transform it into an equivalent standard form problem \(\Pi_2\), with the same optimal cost, and in which the rows of the matrix \(\mathbf A\) are linearly independent. Let \(D_1\) and \(D_2\) be the duals of \(\Pi_1\) and \(\Pi_2\), respectively. Of course the dual problems \(D_1\) and \(D_2\) have the same optimal cost. We have already proved that \(\Pi_2\) and \(D_2\) have the same optimal cost. It follows that \(\Pi_1\) and \(D_1\) have the same optimal cost.

Proof 2

We need a lemma called Farkas’ lemma:

Let \(\mathbf A\) be a matrix of dimensions \(m\times n\) and let \(\mathbf b\) be a vector in \(\mathbb R^m\). Then, exactly one of the following two alternatives holds:

(a) There exists some \(\mathbf x\ge 0\) such that \(\mathbf A\mathbf x=\mathbf b\).

(b) There exists some vector \(\mathbf p\) such that \(\mathbf p'\mathbf A\ge \mathbf 0'\) and \(\mathbf p'\mathbf b < 0\).

Proof. One direction is easy. If there exists some \(\mathbf x\ge 0\) satisfying \(\mathbf A\mathbf x=\mathbf b\), and if \(\mathbf p'\mathbf A\ge 0'\), then \(\mathbf p'\mathbf b=\mathbf p'\mathbf A\mathbf x\ge 0\), which shows that the second alternative cannot hold.

Let us now assume that there exists no vector \(x\ge 0\) satisfying \(\mathbf A\mathbf x=\mathbf b\). Consider the pair of problems

\[\begin{aligned} \text{maximize }~~~&\mathbf 0'\mathbf x&&&\text{minimize }~~~&\mathbf p'\mathbf b\\ \text{subject to}~~~&\mathbf A\mathbf x=\mathbf b&&&\text{subject to }~~~&\mathbf p'\mathbf A\ge \mathbf 0\\&\mathbf x\ge 0 \end{aligned} \]

and note that the first is the dual of the second. The maximization problem is infeasible, which implies that the minimization problem is either unbounded (the optimal cost is \(-\infty\)) or infeasible (by the corollary (a)). Since \(\mathbf p=\mathbf 0\) is a feasible solution to the minimization problem, it follows that the minimization problem is unbounded. Therefore, there exists some \(\mathbf p\) which is feasible, that is, \(\mathbf p'\mathbf A\ge \mathbf 0\), and whose cost is negative, that is, \(\mathbf p'\mathbf b<0\).

If we rephrase Farkas’ Lemma, we get a corollary:

Let \(\mathbf A_1,\cdots,\mathbf A_n\) and \(\mathbf b\) be given vectors and suppose that any vector \(\mathbf p\) that satisfies \(\mathbf p'\mathbf A_i\ge 0\) for all \(i\), must also satisfies \(\mathbf p'\mathbf b\ge 0\). Then, \(\mathbf b\) can be expressed as a nonnegative linear combination of the vectors \(\mathbf A_1,\cdots,\mathbf A_n\).

Now return to the strong duality theorem. We only need to show that there exists a feasible dual variable \(\mathbf p\) that achieves the same loss as the optimal primal solution (by corollary (b)).

Let \(\mathbf x^{*}\) be the optimal solution to the primal problem, and let \(I=\{i\mid \mathbf a_i'\mathbf x^*=b_i\}\) be the set of active constraints. We claim that if \(\mathbf d\) satisfies that \(\mathbf a_i'\mathbf d\ge 0\) for all \(i\in I\), then we have \(\mathbf c'\mathbf d\ge 0\). Otherwise, \(\mathbf d\) is a feasible descent direction, and then \(\mathbf x^{*}\) is not optimal. By Farkas’ Lemma, there exists scalar \(p_i\ge 0\) such that \(\mathbf c=\sum_{i\in I} p_i\mathbf a_i\). If we set \(p_i=0\) for \(i\in [n]-I\), then

\[\mathbf p'\mathbf A=\sum_{i\in I}p_i\mathbf a_i=\mathbf c\\ \mathbf p'\mathbf b=\sum_{i\in I}p_ib_i=\sum_{i\in I}p_i\mathbf a_i'\mathbf x^*=\mathbf c'\mathbf x^* \]

Hence \(\mathbf p\) is the variable we need and the proof is done.